Skip to content

cubeops: propagate database errors from the settings and user getters - #1382

Open
dwin-gharibi wants to merge 9 commits into
TencentCloud:masterfrom
dwin-gharibi:cubeops-store-swallows-db-errors
Open

cubeops: propagate database errors from the settings and user getters#1382
dwin-gharibi wants to merge 9 commits into
TencentCloud:masterfrom
dwin-gharibi:cubeops-store-swallows-db-errors

Conversation

@dwin-gharibi

Copy link
Copy Markdown
Contributor

Closes #1381.

Motivation

GetSystemSetting, GetSetting and GetUserPassword collapsed three different outcomes — row
missing, value empty, and any database error — into ("", nil). Because the scanned value is the
zero value whenever the query fails, val == "" was true for a real error too, so the error was
discarded.

The visible symptom is that a database outage is reported as 401 invalid credentials rather than a
500, which is the opposite of what AuthService.Login's own comment promises. It also made both
branches of if err != nil in Login unreachable dead code.

What this changes

CubeOps/internal/store/setting.go — all three getters now separate the two cases:

if err != nil {
    if errors.Is(err, sql.ErrNoRows) {
        return "", nil
    }
    return "", err
}
return val, nil

The resulting contract is: ("", nil) means absent; a non-nil error means the read failed. An
empty stored value is still reported as absent, which is what every caller already wanted.

CubeOps/internal/service/auth.goLogin and ChangePassword updated for that contract:

  • Login returns 500 for a read failure and ErrInvalidCredentials when the user is absent
    (stored == "") or the password does not match. User enumeration is still not possible: absent
    and wrong-password are indistinguishable to the caller.
  • ChangePassword likewise returns the wrapped error for a read failure and
    ErrInvalidOldPassword when the user is absent or the old password is wrong.

CubeOps/internal/service/auth_test.go — the fakeUserStore returned
("", errors.New("user not found")) for an unknown user, modelling not-found as an error, while
the real store returns ("", nil). That mismatch is why the tests passed against a contract
production never implemented. The fake now returns ("", nil); its assertion (unknown user →
ErrInvalidCredentials) is unchanged and still passes.

Callers that already discard the error (internal/service/openclaw.go:685-701,
internal/service/agenthub.go:457) are untouched — they still get "" and still ignore the error, so
their behaviour is unchanged.

bootstrapMasterKey (internal/store/db.go:83,93) and BootstrapJWTSecret (:139) already had
if err != nil guards that were previously dead; they now actually fire, which is the intended
fail-closed behaviour.

Testing

New: CubeOps/internal/service/auth_db_error_test.go

  • TestLoginSurfacesInfrastructureError — a DB error is wrapped and returned, and is not
    ErrInvalidCredentials.
  • TestLoginUnknownUserStillReportsInvalidCredentials — no enumeration regression.
  • TestChangePasswordSurfacesInfrastructureError — same for the password-change path.
  • TestChangePasswordUnknownUserReportsBadOldPassword — absent user still reports a bad old
    password.
$ cd CubeOps && go test ./...
ok  .../internal/auth        ok  .../internal/config     ok  .../internal/crypto
ok  .../internal/cubemaster  ok  .../internal/handler    ok  .../internal/httputil
ok  .../internal/logging     ok  .../internal/redact     ok  .../internal/server
ok  .../internal/service     ok  .../internal/store
12 packages, 0 failures

CI gates checked locally:

  • gofmt -l ./internal ./cmd — clean (fmt-check).
  • go build ./... — clean.
  • go test ./... — 0 failures (unit-test-checkmake cubeops-test).

Risk / rollout

This is a behaviour change on the login path and is the reason it is split out from #1: during a
database outage, /api/v1/auth/login now returns 500 instead of 401. Any monitoring or client
retry logic keyed on 401 for that case will see 500 instead — which is the correct signal, but
worth calling out.

The UserStore contract change is internal to CubeOps (internal/), so there is no external API
impact.

Copilot AI lite review requested due to automatic review settings August 18, 2026 06:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

if errors.Is(err, sql.ErrNoRows) || val == "" {
return "", nil
if err != nil {
if errors.Is(err, sql.ErrNoRows) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sql.ErrNoRows branch can't fire with the current GORM setup, so the "absent" case is actually detected by the fall-through below.

db.Raw(...).Scan(&val) goes through GORM's Scan (not *sql.Row), and GORM returns a nil error when the result set is empty — with the default &gorm.Config{} (no TranslateError) used in CubeDB/dao/driver/mysql/mysql.go:61, it never surfaces sql.ErrNoRows. That's why the codebase's other getters that genuinely need to detect a missing row use .Row().Scan(...) instead (refresh_token.go:33-35, agenthub.go:229-231). The old code's val == "" clause was the effective check, and here the err == nil path returning the zero value (return val, nil) is what keeps "absent → ("", nil)" working.

This is harmless (behavior is correct) and even a forward-compatible safety net if GORM ever changes, so no change is strictly required — but the PR description presents this branch as the mechanism that separates "absent" from "error", which isn't what happens today. A one-line comment noting the empty-value fall-through is the real path would prevent a future reader from "fixing" it. Same applies to the identical branches in GetSetting and GetUserPassword.

}

func TestChangePasswordSurfacesInfrastructureError(t *testing.T) {
dbDown := errors.New("context canceled")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is weaker than its Login sibling in two ways:

  1. It never asserts the underlying error is preserved. TestLoginSurfacesInfrastructureError checks errors.Is(err, dbDown), but here only err != nil && !errors.Is(err, ErrInvalidOldPassword) is asserted. A regression to a different non-sentinel error that discards the cause (e.g. errors.New("read failed")) would still pass. Consider adding if !errors.Is(err, dbDown) { t.Fatalf(...) } for parity.

  2. errors.New("context canceled") is a misleading stand-in for the "database unreachable" case — a context cancellation is a different failure mode and could mask a wrong reason. The Login test's "dial tcp ...: connection refused" is a better fit.

@cubesandboxbot

cubesandboxbot Bot commented Aug 18, 2026

Copy link
Copy Markdown

Review: PR #1382 — cubeops: propagate database errors from the settings and user getters

AI-generated review; not a human approval.

Overall assessment

This is a well-scoped and correct fix. The root cause is accurately diagnosed: GetSystemSetting, GetSetting, and GetUserPassword collapsed "row missing", "empty value", and "any DB error" into ("", nil), because GORM's Scan into a plain string leaves the target zero-valued when the query fails, so the val == "" fallback masked genuine driver errors. The switch to sql.NullString + Raw().Row().Scan() cleanly separates sql.ErrNoRows (→ absent, ("", nil)) from real failures (→ ("", err)), and the if stored == "" || !VerifyPassword(...) guards in Login/ChangePassword restore the intended contract: absent user and wrong password are indistinguishable (ErrInvalidCredentials / ErrInvalidOldPassword), infrastructure errors are surfaced to the caller.

The handler-layer change (returning a generic "internal server error" and logging the real error server-side) is good security hardening — the 500 paths no longer echo driver text (host/port/DSN) to unauthenticated or authenticated clients. The new unit tests, store contract tests, and the e2e outage suite directly cover the failure modes that previously had no coverage, and the fakeUserStore correction (missing user → ("", nil) instead of an error) aligns the test doubles with the contract production actually implements.

No blocking correctness issues found. The points below are minor.

Findings

  1. internal/auth/handler.go:123 — ChangePassword now masks client-validation errors too. The generic body also applies to service.ChangePassword's validation errors ("oldPassword and newPassword are required", "new password must be at least 4 characters"), which land in this default branch. These were already 500 (pre-existing), but the message previously told the client what to fix; now it is an indistinguishable generic 500. Consider mapping validation errors to 400 or wrapping them in a sentinel.

  2. e2e/db_outage_test.go:84 — the change-password and refresh subtests don't assert 500. They only t.Logf when the status isn't 500, so they pass even if those endpoints return 401/200 during the outage (as long as no driver detail leaks). Both paths deterministically hit the DB while the server is up (GetUserPassword / IsRefreshTokenRevoked) and the auth middleware is signature-only, so asserting 500 here would make the guarantee uniform with the login subtest.

  3. e2e/harness_test.go:179 — the process-crash detection branch never fires. cmd.ProcessState is only populated by Wait(), which is never called in the readiness loop, so the cmd.ProcessState != nil && cmd.ProcessState.Exited() check is always false. If cubeops exits early, the harness polls /health until the 4-minute deadline instead of failing fast. Additionally, do() uses http.DefaultClient with no timeout, so a hung server would stall the test indefinitely.

Non-blocking notes

  • internal/store/db.gobootstrapMasterKey now fail-closes against a DB that lacks t_system_setting. Previously the error was swallowed and the code fell through to t_agenthub_setting; now a missing table aborts startup. The author documents this as intended ("an unmigrated DB … startup fails, as it would anyway at seedDefaultAdmin"), and for a fully unmigrated DB that is true. Worth a final confirmation for the partial-DDL path (CUBE_AUTO_MIGRATION=false with out-of-band DDL that happens to include t_agenthub_setting but not t_system_setting), where the old binary could previously boot off the legacy table.
  • internal/handler/agenthub.go — only writeServiceError was hardened. Many other 500 paths in this file still echo err.Error() to clients (e.g. failed to list instances: ..., failed to update settings: ...). Out of scope for this PR's auth fix, but the same detail-leak pattern exists if a follow-up is planned.
  • The e2e suite is correctly excluded from the default go test ./... gate via the e2e build tag; doc.go keeps the package buildable when the tag is absent.

Testing

The added tests are appropriate and meaningful: store-contract tests exercise the absent-row / driver-error / empty-value tri-state against a real MySQL container; the service tests pin the enumeration-safety and error-propagation behavior; the handler tests verify no driver detail leaks on the three 500 paths; and the e2e outage test validates the wire-level behavior with the database killed mid-run. The auth_test.go fake-store correction is necessary and correct.

Comment thread CubeOps/internal/service/auth.go
Comment thread CubeOps/internal/auth/handler.go
Comment thread CubeOps/internal/store/db.go Outdated
@liciazhu

Copy link
Copy Markdown
Collaborator

Thanks for this PR.
One issue to address before merge:
The same err.Error() leak pattern is still present in the Refresh handler ( handler.go ~line 140):
httputil.WriteError(c, http.StatusInternalServerError, err.Error())
This is the same class of vulnerability the PR addresses for Login and ChangePassword, but the third entry point was missed.
Suggested fix:

  1. Replace err.Error() with "internal server error" on the Refresh 500 path, consistent with the other two handlers.
  2. Add a TestRefreshOutageDoesNotLeakDatabaseDetailsToTheCaller test case to cover it.
    Everything else looks good — the service layer changes, store contract updates, and test coverage are solid. Happy to approve once the Refresh path is brought in line.

… the settings and user getters

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
…proper tests for auth db errors testing

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
…ws really signals an absent row

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
…-vs-error getter contract

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
…n 500 responses

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
…dies leak no database detail

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
…500 paths leaking error detail

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
…fresh 500 body leaks nothing

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
@dwin-gharibi
dwin-gharibi force-pushed the cubeops-store-swallows-db-errors branch from 9b3d06a to 27b92a9 Compare August 19, 2026 08:40
@dwin-gharibi

Copy link
Copy Markdown
Contributor Author

Everything is fine now @liciazhu.

@liciazhu

Copy link
Copy Markdown
Collaborator

LGTM.
Re-reviewed: both prior findings fixed (Refresh + agenthub 500 paths now also scrubbed; store getters distinguish sql.ErrNoRows from real driver errors). Ran full CubeOps suite + race detector locally — all pass.

…ing server and check the 500 bodies

Signed-off-by: Dwin Gharibi <dwin.gharibi@email.kntu.ac.ir>
// Validation errors and DB errors share the 500 path here; finer
// mapping can be added by wrapping with sentinel errors if needed.
httputil.WriteError(c, http.StatusInternalServerError, err.Error())
httputil.WriteError(c, http.StatusInternalServerError, "internal server error")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The switch to a generic body here also masks client-side validation errors, not just DB errors. service.ChangePassword returns plain errors for "oldPassword and newPassword are required" and "new password must be at least 4 characters", which land in this default branch — so a client sending a too-short new password now gets an indistinguishable 500 {"error":"internal server error"} instead of the previous message. The 500 status for these was already pre-existing, but the message at least told the client what to fix. Consider mapping the validation errors to 400 (or wrapping them in a sentinel) so they don't share the DB-failure body — the comment above already notes this conflation.

})

t.Run("refresh does not leak driver detail", func(t *testing.T) {
code, body := do(t, http.MethodPost, inst.baseURL+"/api/v1/auth/refresh", "",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two subtests only t.Logf when the status isn't 500, so they pass even if change-password/refresh return 401 or 200 during the outage (as long as the body carries no driver detail). Both paths deterministically hit the DB while the server is up (GetUserPassword for change-password, IsRefreshTokenRevoked for refresh) and the auth middleware is signature-only, so both should reliably return 500. For consistency with the login subtest above, asserting code == http.StatusInternalServerError here would make the guarantee uniform.

return inst
}
}
if cmd.ProcessState != nil && cmd.ProcessState.Exited() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmd.ProcessState is only populated by Wait(), which is never called in this readiness loop, so cmd.ProcessState != nil is always false here — this crash-detection branch never fires. If cubeops exits early, the harness keeps polling /health until the 4-minute deadline and only then reports the failure via the log tail, instead of failing fast. Also, do() uses http.DefaultClient with no timeout, so if the server hangs during an outage the test stalls indefinitely. A short HTTP timeout (and waiting on the process in a goroutine, or polling cmd.Process liveness) would make the harness fail faster and more reliably.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug Report] CubeOps store layer swallows every database error, so outages surface as "invalid credentials"

3 participants